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In this paper, we describe a simple language for parallel programming. Its semantics is studied thor- 
oughly. The desirable properties of this language and its deficiencies are exhibited by this theoret- 
ical study. Basic results on parallel program schemata are given. We hope in this way to make a case 
for a more formal (i.e. mathematical) approach to the design of languages for systems programming and 
the design of operating systems. 



There is a wide disagreement among systems designers 
as to what are the best primitives for writing sys- 
tems programs. In this paper, we describe a simple 
language for parallel programming and study its 
mathematical properties. 

1. A SIMPLE LANGUAGE FOR PARALLEL PROGRAMMING. 

The features of our mini-language are exhibited on 
the sample program S on fig.l. The conventions are 
close to Algol and we only insist upon the new 
features. The program S consists of a set of decla- 
rations and a body. Variables of type integer* 
channel are declared at line (1), and for any simple 
type o (boolean, real, etc..) we could have decla- 
red a o channel. Then processes f , g and h are 
declared, much like procedures. Aside from usual 
parameters (passed by value in this example, like 
INIT at line (3)), we can declare in the heading of 
the process how it is linked to other processes : at 
line (2) f is stated to communicate via two input 
lines that can carry integers, and one similar out- 
put line. 

The body of a process is an usual Algol program except 
for invocation of wait on an input line (e.g. at (A)) 
or Bend a variable on a line of compatible type 
(e.g. at (5)). The process stays blocked on a wait 
until something is being sent on this line by ano- 
ther process, but nothing can prevent a process 
from performing a send on a line. 
In other words, processes communicate via first-in 
first-out (fifo) queues. 

Calling instances of the processes is done in the 
body of the main program at line (6) where the 
actual names of the channels are bound to the formal 
parameters of the processes. The infix operator par 
initiates the concurrent activation of the processes. 
Such a style of programming is close to may systems 
using EVENT mechanisms (C 1 3 ,[2] ,[3] ,[4]) . A picto- 
rial representation of the program is the schema P 
on fig. 2., where the nodes represent processes and 
the arcs communication channels between these pro- 
cesses. 

What sort of things would we like to prove on a 
program like S ? Firstly, that all processes in S 
run forever. Secondly, more precisely, that S prints 
out (at line (7)) an alternating sequence of O's 
and l's forever. Third, that if one of the processes 
were to stop at some time for an extraneous reason, 
the whole system would stop. 

The ability to state formally this kind of property 
of a parallel program and to prove them within a 
formal logical framework is the central motivation 
for the theoretical study of the next sections. 

2. PARALLEL COMPUTATION. 

Informally speaking, a parallel computation is orga- 
nized in the following way : some autonomous compu- 
ting stations are connected to each other in a net- 
work by communication lines. Computing stations 
exchange information through these lines. A given 
station computes on data coming along its input lines, 



Begin 

(1) Integer channel X, Y, Z, Tl, T2 ; 

(2) Recess f (integer in U,V; integer out W) ; 

Begin integer I ; logical B ; 
B true ; 
Repeat Begin 

(4) I 2- if B then wait(M) else vaitQJ) ; 
(7) print (I) ; 

(5) send I on W ; 
B :« — lB ; 
end ; 

End ; 

Pi*ocess giinteger in U ; integer out V, W) ; 
Begin integer I ; logical B ; 
B true ; 
Repeat Begin 
I :» wait (U) ; 

if B then send I on V else send I on W ; 
B IB ; 

End ; 
End ; 

(3) Process h(integer in M\integer out V; integer INIT); 

Begin integer I ; 
send INIT on V ; 
Repeat Begin 
I wait(U) ; 
send I on V ; 
End ; 
End ; 

Cament : body of mainprogram ; 

(6) f(Y,Z,X) par g(X,Tl,T2) par h(Tl ,Y,0) par h(T2,Z,i; 
End ; 

Fig.l. Sample parallel program S. 




Fig. 2. The schema P for the program S. 
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using some memory of its own, to produce output on 
some or all of its output lines. It is assumed that : 
i) Communication lines are the only way by which 
computing stations may communicate, 
ii) A communication line transmits information within 
an impredic table but finite amount of time. 
Restrictions are Imposed on tho bohnviour of compu- 
ting stations : 

iii) At any given time, a computing station is either 
computing or waiting for information on one of 
its input lines, 
iv) Each computing station follows a sequential 
program. (We call here sequential program what is 
usually called a program elsewhere) . 
Remarks : first, since several computing stations 
may be computing simultaneously, this model indeed 
exhibits some form of parallelism. Second, restric- 
tion iii) means that a computing station cannot be 
waiting on data coming from one or another of its 
input lines, or alternately that no two computing 
stations are allowed to eend data on the same charjiel 
Third, we do not restrict the computing stations to 
have a finite memory. 

The reader who is mathematically inclined can think 
of a set of Turing machines connected via-one-way 
tapes , where each machine can use its own working 
tape. 

We formalize now the notion of parallel computation 
introduced above. 

2 • 1 • s y ntax 

A parallel program schema is an oriented graph with 
labeled nodes and edges, together with some supple- 
mentary edges (see fig. 3.) : incoming edges with 
only end vertices, meant to represent the input lines, 
and cutcoraing edges, with only origin vertices, the 
. output lines. 
2.2. Semantics 

2.2.1. Outline 

Edges in a schema are thought of as pipes : each 
edge is able to carry data of a given type D(e.g : 
integer, boolean, pointer, procedure etc... ). 
An observer placed on the line witnesses its traffic, 
a (possibly infinite) sequence of objects of type D : 
it is called the history of the line. Since a compu- 
ting station has its own memory, it is not a partial 
function from the domains of the inputs into the 
domain of the outputs, but rather a function from the 
histories' of its input lines into the histories of 
its output lines. ,.. 

2.2.2. Sequence domains 

Let D? i.be the set of finite or denumerably infinite 
sequences of elements over a set D. In D w we include 
the. empty sequence A. The relation c defined by 

X»£:Y^ iff X; is an initial segment of Y 
is 'a; partial order on D w .' The minjmal element of D W 
is A. T Any increasing chain £ in D : 
X. c x,c. V.c xv. 



we calf-'lim £ 
(c.p.o)^;D w 



£ ... hjis a least upper bound which 
Hence D is a complete partia l order 



2.2.3.f Domain of interpretation 
To each "edge e-in-a^schema, we associate a set D , the 
type of <the : objects^! t may carpr. The history of e 
line ■%* is then "an* element of D • f 

- , 2. 2*. 4. ^Continuous mappings . 

A mapping :f;;fromi a complete partial order A irto a 
complete. partial order. B, is continuous iff, for any 
increasing chain a of A 

f (lim a) - lim'^f (a) 
A B ' ' 

Note that a continuous, mapping is also monotonic, i.e. 
X £ y •> f(x),'£ f(y) 

The following -mappings (for fzret), R (for remainder)' 
and i4(f or append) are examples of continuous mappings : 

- F : to any sequence x in D W , F associates the 
(unit length) sequence constituted of the lcttmost 
element of x. 




Fig. 3. A parallel program schema. 




Fig. 4. The complete partial order D f when D-U^aj-) 



- R : to any sequence x in D , R associate the 
sequence of the right of its leftmost element. 

- A : takes two arguments LI and L2 in D w to pro- 
duce the sequence : (leftmost element of LI) follo- 
wed by L2. 

More precisely, F ,/?, and A obey the axioms : 
1) tf(A) - A 2) 4(A,X) - A 3) 4(X,A) - F(X) 
4) F04(X,Y) - F(X) 5) A (F(X) ,fl(X) ) - X 
6) X • A v /?G*(X,Y)) - Y 

For properties such as deadlock, we shall need to 
talk formally about the length of a sequence. An 
elegant way to do so within our formalism is to take 
the integers with their usual order and complete 
them with an extra element - to obtain the complete 
partial order N (fig. 5). 

The mapping length from D u to N which maps a sequen- 
ce into its length is continuous ; note also that 
addition in N is continuous. 

2.2.5. Computing stations 

We are now ready to interpret the nodes in a parallel 
schema. To each node with input lines carrying data 
in Dj,D2» . .., D n and producing data in D^DJ,. 

we associate p continuous functions from 

x x ... D~ into (respectively) .D j^D^D* 4 " 

For example, in fig. 6, we specify two continuous 
functions f ^ .and f 2 in order to interpret node f 



£ 2 : *l x D° x + D'f 



Fig. 5. The c.p.o "N. 
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Examples : 

The process f of program. 5? is associated to the 
continuous function f in N u x N W 4 N w defined 
recursively by : 

f(U.V) - >4(F(U),>I(F(V),f(/?(U) f /?(V)))). 

The process g is associated to two functions, one per 

outputline, defined recursively by : 

g.(U) -4<F(U) f g .(/?(tf(U)))) and 

g*(U) - ^(F(ff(uj) lg2 (/?</?cu)))). 

Similarly the function h maps N w into n" : 
h(U,x) - 4({x},U) where x is in N and the notation 
{x} means the unit length sequence whose first 
element is x. 

In these examples, not much computation is actually 
performed on the inputs, but an arbitrary amount of 
computation could be performed by a computing station, 
requiring possibly an unbounded amount of memory. 
The restriction of the interpretation of nodes to 
continuous functions can be understood in concrete 
terms : 

a) Mono tonicity means that receiving more input at 
a computing station can only provoke it to send more 
output. Indeed this a crucial property since it 
allows parallel operation : a machine need not have 
all of its input to start computing, since future 
input concerns only future output. 

b) Furthermore continuity prevents any station 
from deciding to send some output only after it has 
received an infinite amount of input. 

Any process written in the simple programming langua- 
ge of section 1. corresponds to a set of continuous 
functions. The recursive definition of these func - 
tions is obtained by the usual method of McCarthy for 
converting flow-chart programs to recursive defini- 
tions . 

3. FIXPOINT EQUATIONS. 

Rather than studying the behaviour of a complex ma- 
chine, we want to study the properties of the solu- 
tion of a set of equations. To each parallel program 



Example 



(i.e. interpreted schema) we associate a setj^ of 
equations on sequence domains in such a way that a 
set of sequences is a possible solution of the sys- 
tem iff it is a possible set of histories for the 
conmunica tion' s lines of the program : 

i) To every line c, of type D , associate a 
variable X ranging over D e . 
ii) If Xj,X 2 , X^ are the variables associated 



to the input lines and i,, 



i k are the se- 



quences fed as inputs on the lines include the 
equations : 

i , 



1 



iii) For each node f, interpreted with the functions 

f., ...f , with input variables X.,..., X 

1 P I* • n 

output variables Xj, ... X* include p equations 



inZ^ 



X' 

p 



£,(x,...x a ) 



••V 



Clearly, the histories of the Unes of the program P 
have to satisfy the system X.X is a set of fix- 
point equations over c.p.o.'s, where the operators 
are continuous. It is a well-known mathematical result 
(see for example Milner [63) that such a system admits 
a u nique minimal solution . It is outside the scope of 
this paper to show that this minimal solution consti- 
tutes indeed the vector of histories of the communi- 
cation's lines, given a suitable implementation. Such 
a proof can be found in Cadiou [5] in a similar set 
up. 

The first property of this minimal solution gives us 
access to the most powerful rule of induction used 




x . 


■ i 


X 2 


- f(x,,x 3 ) 


X 5 


-g,(x 2 ) 


h 


- 8 2 (X 2 ) 


*7 


- vv 


X 6 


- k 2 tX 5 ) 


X 8 




X 3 


- VW 



fig .7. The program P and the associated system 3L 



in proving programs correct (see Manna, Ness, 
Vuillemin [9]), i.e. Scott's rule : 

Property 1 [Kleene] 

The minimal solution (Y^) ,Y(X ; ) . . Y(X^)} of the 
system E^- {X . - T.(X | t ...X^) 1 i c [l.n]} where 
the are terms built out of continuous operators 
is lim (X* , ... X 1 ) where 

X? « A(i g [ 1,n1) (Strictly speaking there 

; + l * .might be n different A's) 

X i * T i (X ) » X u > <* € n,n]). 

Scott's induction rule in this case can be stated ' 
as follows, if P is an admissible predicate (see 
Manna, Ness, Vuillemin [9]) : 
. P(A,...A) 

P( X| , ... x) sP^ix....!), ...t JBuiAU. 



PCYCXj), 



.. Ytt n » 



A property of a parallel program is stated as a 
relation between the input sequences and the output 
sequences or in general between the histories of 
some communication lines. Since we may use Scott's 
rule, all the techniques for proving properties of 
recursive programs studied in Vuillemin [10] are 
available, in particular structural induction and 
recursion induction. 

Example : The system T associated with program S 
is : * 

X - f(Y,Z) 

V - h(T r 0) 

Z > MT 2 ,») 

V g,(X) 

V *2 (X) 

where f, g |f g 2 and h are given in §2.2.5. 

As an illustration, let us prove that the history X, 
which is exactly what S prints out, is an infinite 
alternating sequence of O's and I's. In other words, 
if X is the minimal fixpoint of I- A({0},A({ 1), X)) . 
then X - X 

The system X can be reduced to a single fixpoint 
equation : 

x - f(h( gl (x), o),h(g 2 (x),i)) (i) 

Using the definition of f and h, and the properties 
of F and A we transform eq.(l) to 

X « .4({0}, A< s (\) 9 f(g,(X), g 2 (X)))) (2) 



Lemma : 
Proof 



For all U , U - f (g (U),g (U)) 



By structural induction. The lemma is obviously true 
for A, and for any sequence of length J. Assume 
it is true for V, then : 



474 



Format Models of Parallel Computation 



f(g 1 (>5({a} > i4({b},V))) > g 2 (>4({a},>l({b},V))) 

- ^£{a} > /l({b} r f(g 1 (V),g 2 (V)))) 

- A({a),A (b},V))' by induction hypothesis. [] 
From eq.(2) and the lemma above we deduce : 

X c X. 

With this Lemma again' it is trivial to see that 
X c X, which proves the result. Since the mapping 
length is continuous, length (X) is the minimal 
solution (in N) of 

length(X) - 2 + length(X) 
which is obviously Hence T. and T 2 are infinite 
sequences and so are Y and Z. We have thus answered 
the first two questions raised in sectionl . about 
program S. □ 

The simplicity of the program S and the proof produ- 
ced should not induce the reader into believing that 
only very simple minded proofs are feasible. Milner 
and Weyrauch [7] used the system LCF, based on Scott's 
induction rule, to check mechanically the complete 
proof of the correctness of a small compiler, a very 
large proof indeed. LCF can be readily used for on r 
purposes and very large and trustworthy proofs cou ! i 
be produced on this system. 

Property 2 [Scott] 

is a continuous function 



of the 


parameters of 


the iystera, 


in 


particular the 


values 


of the input i 


streams, or 


the 


operators of the 



system . 

In more concrete terms, Property 2 means that, in this 
model of parallel computation : 

1 . Arbitrary interconnection of systems, as well as 
processes, is legitimate. Hence, top-down design finds 
here a mathematical justification since we can post- 
pone the decision to implement a given function by a 
single process or a set of interconnected processes : 
this decision will not introduce perturbations in the 
remainder of the system. 

< 2. A parallel program can be safely simulated on a 
sequential machine, provided the scheduling algorithm 
is fair enough, i.e. it eventually attributes some 
more computing time to a process which wants it. If 
this . algorithm is not fair however, the only thing 
that may happen is for the parallel program to pro- 
duce less output than what could be expected . But 
what is produced is correct. 

This remark and a -simple argument on lengths answer 
the last question about program S raised in the first 
section'. 

A. RECURSION 



we associate tne systemi 



g 2 (F(f(i,X))) 



gl (F(f(i f X))> 



(N.S. : this is a way to ensure that the parallel 
recursive programs are syntactically well formed. ; 
it is sufficient for our purposes although it may 
give several labels to an edge). We construct now a 
set of fixpoint equations that contain variables in 
two types : sequence domains, and continuous mapping! 
between sequence domains. 

Example : 

To the schema on fig. 8. 
o - F(i) 
X - 

where X and F are respectively an. unknown sequence 
and an unknown continuous mapping between sequence 
domains* The continuous mappings from a c.p.o into 
a c.p.o constitute also a c.p.o. with the ordering 
f c g iff Vx f (x) c g(x) 

The existence of a minimal (now functional) solution 
is still assured and Property 1 and Property 2 hold 
along with their concrete interpretation. A little 
bit more care has to be exerted to make sure that 
the implementation computes the minimal fixpoint. 
The only problem is to know when to start unfolding 
a recursive call to a process. The good strategy is 
not to start when input is presented but when output 
is requested. This rule is basically the delay rule 
of Vuillemin L 10]. 

5. SC HEMATOLOGY. 

Structural properties of parallel programs' are dis- 
covered in studying parallel program schemata. For 
example we can prove that the schemata on fig. 9. 
are equivalent, i.e. whatever process f and g may be 
the two resulting programs will be equivalent. 





Fig. 9. Two equivalent schemata. 



The parallel programs introduced so far actually 
exhibit a bounded parallelism : only a finite number 
of processes may compute simultaneously. It is 
necessary and easy to introduce the recursive paral- 
lel programs, where an unbounded number of machines 
may compute in parallel. 

A recursive parallel schema is a set F j» F 2 » ••• F i 
of parallel schemata in which some nodes may be 



-If a parallel schema Fj has 



labeled F^F^ .. Fj 

ir.put lines labeled ij,i 2> and output lin*s 

°l» 0 2»' ,,0 g» ther.'in each occurrence of F^ the s?je 
labels must occur on its input and output lines. An 
example is given on fig. 8. 




Fig .8. A recursive parallel schema. 



(Nota : these schemata are partially interpreted : 
the node f?")called a 2-piicator sends a copy of 
each input~on each output line. We allow such nodes 
in schemata because they introduce no new fixpoint 
equations) . 

We state the main results (Courcelle, Kahn, 
Vuillemin [ 11]) : 
Theorem 1 : The equivalence of schemata containing 
uninterpreted processes and n-plicators 
is decidable. 
Theorem 2 : There exists a unique minimal schema S 
(i.e. containing a minimum number of 
process nodes) equivalent to a given 
schema S . 

Theorem 3 : The systems of equations corresponding 
to S containing the minimum number of 
equations are obtained by taking minimal 
cuts of S. 

The results concerning recursive parallel schemata 
are much harder. Restricting ourselves to recursive 
processes with one input and one input we know 
(Courcelle-Vuillemin [12]) : 

Theorem 4 : Equivalence of recursive parallel sche- 
mata is decidable. 
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6. DISCUSSION AND CONCLUSION 

The kind of parallel programming we have studied in 
this paper is severely limited : it can produce only?.. 
determinate programs. We argue however that : 

i) large parts of operating systems are written so 
as to be determinate. The method of monitors 
advocated by Hoare narrows down the possible 
locations of non-determinacy . 
ii) the primitives wait and send x on y that we 

studied are not too far from reality as exempli- 
fied by El3,E33,E43. 
iii) We do not think it is impossible to extend the 
theory to non-determinate parallel programs, 
although how to satisfactorily do so is far 
from obvious. 

iv) The programming language we have introduced can- 
be extended by adding hew primitive processes 
(i.e. that cannot be programmed as processes 
with wait and send), A typical such process is 
WARN {integer in X,Y ; logical out Z) that 
sends a true value on its output line each time 
some integer is received on either of its input 
lines. The only condition to be verified by the 
new primitive processes, and verified by WARN, 
is that the history of the output line be a 
continuous function of the histories of the 
input lines. 

Looking now at the merits of our approach, we see the 
essential one as the eradication of the notion of 
state of a complex system . More precisely, in Lauer 
t 13 □ and Gilbert [14J for example, a system is thought 
of as having a huge "state vector" and making nonde- 
terministic transitions from state to state. This 
view leads to proofs growing exponentially with the 
number of processes (we grow linearly) and is blind 
to the structure of the system, making the proofs 
counter-intuitive. Furthermpre is cannot deal with 
an unbounded number of processes, something we get 
almost "for free". Our proofs can be checked mecha- 
nically in LCF [8], another non negligible advantage 
since they will often be tedious but without great 
mathematical depth. 

(kir last conclusion is to recall a principle that has 
been so often fruitful in Computer Science and that 
is central to Scott's theory of computation : a 
good concept is one that is closed 

1. under arbitrary composition 

2. under recursion. 
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